Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#820: create simple regression test suite #823

Merged
merged 34 commits into from
Sep 13, 2024
Merged

#820: create simple regression test suite #823

merged 34 commits into from
Sep 13, 2024

Conversation

mmerdes
Copy link
Collaborator

@mmerdes mmerdes commented Sep 10, 2024 •
edited
Loading

Description

This PR adds a regression test suite based on hurl.

Corresponding issue

Closes #820

New or changed dependencies

  • In order to run the tests, the hurl CLI tool must be installed.
  • no new Python dependencies

Checklist

  • I have updated my branch to main (e.g. through git rebase main)
  • My code follows the style guide and was checked with pre-commit before committing
  • I have commented my code
  • [ ] I have added sufficient unit and integration tests
  • I have updated the CHANGELOG.md

Please check all finished tasks. If some tasks do not apply to your PR, please cross their text out (by using ~...~) and remove their checkboxes.

mmerdes added 22 commits September 3, 2024 17:34
@mmerdes
test: add first hurl tests for currentness and mapping saturation ind…
e649155 
…icators # 820

#820
@mmerdes
test: fix naming for currentness test and its payload # 820
3d1ccf6 
#820
@mmerdes
test: fix naming for mapping saturation test and its payload # 820
b6de411 
#820
@mmerdes
test: simplify naming #820
13e1ecf 
#820
@mmerdes
test: add test for metadata endpoint and run-script #820
ba0c202 
#820
@mmerdes
test: add test with topic and area such that indicator cannot be crea…
f4cefc3 
…ted #820

#820
@mmerdes
test: add test for road comparison #820
facf38f 
#820
@mmerdes
test: add test for building comparison #820
858daf8 
#820
@mmerdes
test: adjust volatile test #820
bc7e80e 
#820

Co-authored-by:
@mmerdes
test: add longer-running test for road currentnes #820
a647f01 
#820
@mmerdes
test: improve check for content length #820
ec259ab 
#820
@mmerdes
test: add regexp matching alternative for description check #820
f0f1386 
#820
@mmerdes
test: add test for completeness of attribute 'address_housenumber' fo…
d24cfba 
…r building count #820

#820
@mmerdes
refactor: fix naming issue #820
8ec9837 
#820
@mmerdes
test: add test for completeness of attribute 'maxspeed' for roads #820
f890d0b 
#820
@mmerdes
refactor: improve naming #820
d62537b 
#820
@mmerdes
refactor: extract base_url variable from all test to env variable #820
b9cee01 
#820
@mmerdes
feat: add per-stage run scripts: TEST #820
d17844b 
#820
@mmerdes
feat: add per-stage run scripts: PROD #820
e4b8d9c 
#820
@mmerdes
feat: add per-stage run scripts: DEV #820
be37f81 
#820
@mmerdes
doc: add README with intro to hurl testing #820
63e5553 
#820
@mmerdes
doc: describe regression tests in development docs #820
ceb4458 
#820
@mmerdes mmerdes added the tests Issues with tests label Sep 10, 2024
@mmerdes mmerdes linked an issue Sep 10, 2024 that may be closed by this pull request
Create simple regression test suite #820
Closed
@mmerdes mmerdes requested a review from joker234 September 11, 2024 07:54
joker234
joker234 reviewed Sep 11, 2024
View reviewed changes
Copy link
Member

@joker234 joker234 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the PR!

some remarks:

  • env variables should always be upper case: https://bertvv.github.io/cheat-sheets/Bash.html#variables
  • I suggest to change the directory to the directory the tests are, otherwise you delete the folders somewhere else or the scripts are not executing properly
  • I suggested a OS detection for the report opening in the end
  • always state which shell should execute the script to avoid shell specific errors on systems with different default shells

docs/development_setup.md Outdated Show resolved Hide resolved
regression-tests/buildingarea_adminarea_buildingcomparison.hurl Outdated Show resolved Hide resolved
regression-tests/long-running/roads_bbox_currentness.hurl Outdated Show resolved Hide resolved
regression-tests/buildingcount_bbox_attributecompleteness.hurl Outdated Show resolved Hide resolved
regression-tests/buildingcount_bbox_currentness.hurl Outdated Show resolved Hide resolved
regression-tests/__run_hurl_tests_for_stage.sh Outdated Show resolved Hide resolved
regression-tests/__run_hurl_tests_for_stage.sh Outdated Show resolved Hide resolved
regression-tests/run_hurl_tests_DEV.sh Show resolved Hide resolved
regression-tests/run_hurl_tests_PROD.sh Show resolved Hide resolved
regression-tests/run_hurl_tests_TEST.sh Show resolved Hide resolved
@mmerdes mmerdes self-assigned this Sep 11, 2024
mmerdes and others added 2 commits September 11, 2024 15:03
@mmerdes @joker234
Incorporate review feedback # 820
ee78ede 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes @joker234
Incorporate review feedback # 820
f42714a 
Co-authored-by: Johannes Visintini <[email protected]>
mmerdes and others added 8 commits September 11, 2024 15:06
@mmerdes @joker234
Incorporate review feedback # 820
a16d040 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes @joker234
Incorporate review feedback # 820
59fb901 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes @joker234
Incorporate review feedback # 820
1849686 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes @joker234
Update regression-tests/__run_hurl_tests_for_stage.sh
0116cf8 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes @joker234
Incorporate review feedback # 820
74ea4be 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes @joker234
Incorporate review feedback # 820
3646b8c 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes @joker234
Incorporate review feedback # 820
0068fc4 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes @joker234
Incorporate review feedback # 820
1194048 
Co-authored-by: Johannes Visintini <[email protected]>
@mmerdes
Copy link
Collaborator Author

mmerdes commented Sep 11, 2024

@joker234
Thanks for the review comments - good to have a scripting expert around ;)

@mmerdes mmerdes merged commit 5b6135a into main Sep 13, 2024
2 checks passed
@mmerdes mmerdes deleted the 820-create-simple-regression-test-suite branch September 13, 2024 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joker234 joker234 joker234 approved these changes

Assignees

@mmerdes mmerdes

Labels
tests Issues with tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create simple regression test suite
2 participants
@mmerdes @joker234